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FOREWORD 


This  research  and  development  was  conducted  in  support  of  Navy  Decision  Coordinat¬ 
ing  Paper  Z0789-PN,  Subproject  .01  (Class  "A"  Electronic  Equipment  Maintenance 
Training  (EEMT)  System)  and  was  sponsored  by  the  Chief  of  Naval  Operations  (OP-115  and 
OP-987H)  and  the  Chief  of  Naval  Material  (NMAT-08EI). 

This  report  is  the  third  in  a  series  concerning  the  EEMT  project,  which  has  been 
divided  into  four  phases:  (1)  concept  formulation,  (2)  system  definition,  (3)  prototype 
development,  and  (4)  system  test  and  evaluation  (T&E).  Previous  reports  have  described 
Phases  I  and  II  (NPRDC  TN  79-3  and  NPRDC  TU  81-11).  This  report  forwards  the  EEMT 
Device  Test  and  Evaluation  Master  Plan  (DTEMP),  which  is  a  product  of  Phase  II  and  was 
prepared  under  contract  N00123r78-C-0925  to  Honeywell  Systems  and  Research  Center. 
The  DTEMP  has  been  approved  by  CNOTO'P-115)  and  will  be  implemented  by  the  Training 
Analysis  and  Evaluation  group  (1AEG),  Orlando,  FL  commencing  in  FY  1981.  Subsequent 
reports  to  be  published  concerning  Phase  n  will  address  EEMT  life  cycle  costs  and  the 
T«5cE  syllabus  documents. 

Two  other  reports  have  been  published  on  a  related  research  project.  These  reports 
described  the  field  evaluation  of  and  hardware/software  development  for  the  Generalized 
Maintenance  Trainer  Simulater  (NPRDC  TRs  80-30  and  81-9). 

Appreciation  is  extended  to  Mr.  Ted  Pearson,  Dr.  Lyle  MacKeraghan,  and  Mr. 
William  Stubbs  of  TAEG,  who  provided  their  time  and  ta  -ts  during  the  review  and  final 
preparation  phases. 

The  contracting  officer's  technical  representative  was  Dr.  Tames  S.  McMichael. 


JAMES  F.  KELLY,  OR  JAMES  J.  REGAN 

Commanding  Officer  Technical  Director 
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INTRODUCTION 


Problem 

The  use  of  operational  equipment  as  a  primary  means  for  providing  training  and  q 
hands-on  practice  of  electronics  maintenance  tasks  in  electronics-oriented  classr"A"^~ 
schools  can  be  costly,  inefficient,  dangerous,  and  inadequate.  Therefore,  the  Navy  has 
initiated  an  engineering  development  effort  to  develop,  test,  and  evaluate  a  general- 
purpose  electronic  equipment  maintenance  training  (EEMT)  system  that  can  significantly 
reduce  reliance  on  operational  equipment  at  the  clas^A"  school  level.  Planning  for 
system  test  and  evaluation  (T<5cE)  is  required  in  support  of  this  effort. 


Background 

The  EEMT  project  was  initiated  in  response  to  Operational  Requirement  (OR)  PN-53 
of  26  Duly  1976  and  is  being  conducted  in  support  of  Navy  Decision  Coordinating  Paper 
(NDCP)  Z0789-PN.01.  The  OR  recognized  the  deficiencies  associated  with  the  use  of 
operational  prime  equipment  at  the  "A"  school  level  and  established  performance  ?\  ils 
and  general  criteria  to  be  met  by  the  training  system.  The  EEMT  project  was  divided  i  to 
four  phases:  (1)  concept  formulation,  (2)  system  definition,  (3)  prototype  developme  , 
and  (4)  system  test  and  evaluation  (T&E). 


In  Phase  I,  which  involved  the  completion  of  training  pipeline  and  preliminary  cost 
analyses,  concept  feasibility  was  affirmed  and  a  basic  system  design  concept  was 
formulated  (Wylie  <5c  Bailey,  1978).  The  Electronics  Technician  (ET)  and  Electronic 
Warfare  Technician  (EW)  Class  "A"  schools  were  designate!  as  the  target  users  of  the 
prototype  EEMT. 

In  Phase  II,  it  was  demonstrated  that  a  great  deal  of  corn  onality  exists  in  "A"  school 
curriculum  for  the  ET  and  EW  ratings  and  that  it  was  t  :asible  to  implement  this 
curriculum  using  the  EEMT  (Pine,  Koch,  <5c  Malec,  1981).  'i.  e  design  for  the  prototype 
EEMT  was  selected  and  specifications  were  prepared  for  tw  fimensiona1  (2D)  and  three- 
dimensional  (3D)  subsystems.  Also,  input  for  the  EFM<  Device  Test  and  Evaluation 
Master  Plan  (DTEMP),  which  will  govern  EEMT  T<5cE  do-  Pr.iv  III  (system  definition) 
and  beyond,  was  developed.  ,  >  ,.J 
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The  purpose  of  the  effort  described  herein  was  i-  v.:  j  .  e  EEMT  DTEMP/  This 

plan  will  be  used  to  (l)  direct  and  control  all  T&E  phases,  .2)  identify  contractor  and 
government  resources,  long-range  planning  progrummi  ,1  budgeting  required  to 
support  the  T&E  effort. 


APPRCncH 

The  preparation  of  the  EEMT  DTEMP  was  guided  by  OPNAV  Instruction  3960.10  of  22 
October  1975,  which  (1)  establishes  policy  for  test  and  evaluation  (T3cE)  in  Navy 
acquisition  programs,  (2)  defines  developing  agency  responsibilities  for  planning,  conduct¬ 
ing,  and  reporting  T&E  activities,  and  (3)  provides  the  basic  guidelines  and  format  for 
preparing  T&E  plans.  The  EEMT  DTEMP  was  tailored  to  apply  to  a  training  system  and 
was  designed  to  provide  for  two  types  of  testing:  (1)  device  test  and  evaluation  (DT&E) 
and  (2)  training  capabilities  testing  (TCT).  Also,  it  was  to  provide  for  testing  at  the 
various  systems  acquisition  stages. 


DT&E  is  intended  to  focus  on  certain  develo;  mental  issues  involving  the  demonstra¬ 
tion  of  2D  and  3D  system  design  features  and  to  provide  for  testing  various  training  and 
component  hardware/software  performance  capabilities.  This  will  include  measures  of 
system  reliability,  maintainability,  and  supportability  characteristics.  Also,  it  is  to 
provide  for  phased  testing  of  2D,  3D,  and  combined  2D/3D  trainer  capabilities  as 
appropriate  within  the  context  of  the  hardware  system  design.  These  tests  are  to  be 
conducted  both  in  the  contractor's  facilities  and  in  the  class  "A"  school  laboratory 
environment. 

TCT  involves  the  experimental  evaluation  of  both  EEMT  performance  and  trainee 
achievement  when  the  trainer  is  employed  in  the  class  "A"  school.  Trainee  performance  is 
to  be  assessed  in  accordance  with  the  design  goals  and  objectives  specified  for  the  entire 
EEMT  (trainer,  teachware,  and  supportive  materials).  The  TCT  is  to  be  conducted  in  two 
phases  at  the  ET  "A"  school,  Service  School  Command,  Great  Lakes,  1L  and  the 
Consolidated  Naval  Electronics  Warfare  School  (CNEWS),  Pensacola,  FL.  Each  phase  will 
employ  identical  measurement  criteria  and  test  procedures.  However,  they  may  differ, 
depending  on  the  specific  operational  hardware  characteristics  included  in  the  EEMT 
teachware  and  3D  component. 


RESULTS 

A  DTEMP  was  prepared  that  is  consistent  with  policies  and  procedures  established  in 
OPNAV  Instruction  3960.10  and  that  addresses  specific  T&E  issues  outlined  in  OR  PN-50 
and  NDCP  Z-0789-PN.  A  copy  of  this  plan  is  provided  in  the  appendix. 

The  DTEMP  incorporates  measures  of  both  hardware  and  training  system  perfor¬ 
mance.  It  has  been  reviewed  and  approved  for  implementation  by  the  Chief  of  Naval 
Operations  (OP-115).  Implementation,  commencing  in  FY  1981,  will  be  directed  by  the 
Training  Analysis  and  Evaluation  Group  (TAEG),  Orlando,  in  accordance  with  details  and 
schedules  contained  in  the  DTEMP. 


CONCLUSION 

The  DTEMP  provides  for  a  comprehensive  test  of  EEMT  hardware  and  training  system 
capabilities.  Programmatic  implementation  should  provide  appropriate  measures  of 
system  reliability,  supportability,  and  suitability  for  service  use. 


FUTURE  DIRECTION 

Planning  for  T&E  implementation  is  currently  underway.  The  first  two  EEMT  2D 
units  will  be  delivered  to  NAVPERSRANDCEN  in  June  1981  and  will  be  used  primarily  for 
demonstration,  lessonware/software  review,  and  support  of  on-going  research.  Ten  2D 
units  will  be  delivered  to  the  ET  "A"  school,  Great  Lakes,  in  September  1981;  and  eight,  to 
CNEWS  Pensacola,  in  December  1981.  2D  unit  DT&E  will  commence  following  delivery 
and  installation  and  will  continue  through  the  second  quarter  of  1983.  One  3D  unit  will  be 
delivered  to  each  test  site  in  April  1982  and  will  be  integrated  into  the  T&F.  program. 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN 

Electronic  Equipment  Maintenance  Training  System  (Device  1 1 B 106) 


1.  Full  Program  Title: 


Part  I 

Administrative  Information 


Class  "A"  Electronic  Equipment 
Maintenance  Training  System 


2.  Program  Element  No.  647Q3N  Project  No.  Z07S9-PN 


S&TO  No. _ OR  No.  OR-PN5Q 

DP  No.  DP-PNI50 


X  Acquisition  Category:  III  DA:  NAVPERSRANDCEN 

4.  Points  of  Contact: 


TITLE 

NAME 

OPNAV  Mission  Sponsor 

M.  K.  Malehcrn 

Project  Manager/AM 

V.  M.  Malec 

R&D  Coordinator 

M 

Dr.  R.  Smith 

W 

NAVMAT  Coordinator 

M.  Chen 

DA  Test  Director 

| 

V.  M.  Malec 

OT  Director 

C.  Cordell 

CODE  PHONE  NO. 


OP-102 

AV:  224-5016 
(202) 694-4898 

NAVPERSRANDCEN 

AV:  933-7122 
(714)  225-7122 

OP -987 H 

AV:  227-9694 
(202)  697-9694 

NMAT  08E1 

AV:  222-9013 
(202)  692-9013 

NAVPERSRANDCEN 

AV:  933-7122 
(714)  225-7122 

TAEG 

AV:  791-4367 
(305)  646-4367 

DEVICE  TEST  AND  EVALUATION  MASTER  PLAN 

Electronic  Equipment  Maintenance  Training  System  (Device  11B106) 


5.  Funding  and  Procurement: 


a.  Funding 

RDT<5cE 

Procurement 

b.  Procurement  Quantity 

Preproduction  Prototype 
Production 

6.  Delivery  and  Installation: 


Fiscal  Year  ($  in  millions) 

S3  84  85 


79  <Jc 

Prior  80  81  82 


1.660  1.061  .993  .757  .101 


20 


TBD  TBD  TBD 


Fiscal  Year  (No.  of  Units) 


80  81  82 


83 


84  85 


a.  Delivery  Schedule 

Preproduction  Prototype  20 1 

Production  TBD  TBD  TBD 

b.  Programmed  Installation 

FY  81/82:  EW  "A"  School,  NAVTECHTRACEN,  Pensacola,  FL,  and  ET  "A"  School, 
Great  Lakes,  IL  FY  83,  84,  85:  TBD. 

7.  Remarks:  None. 


‘Student  station--A  student  station  consists  of  a  2D  and  3D  capability  with 
peripherals.  There  will  be  20  prototype  2D  units  and  2  prototype  3D  units  that  interface 
with  2D  units. 
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DEVICE  TEST  AND  EVALUATION  MASTER  PLAN 

Electronic  Equipment  Maintenance  Training  System  (Device  11B1Q6) 


Part  II 
Description 


1.  System  Description  and  Mission 

a.  Training  System  Description.  The  electronic  equipment  maintenance  training 
(EEMT)  system  will  be  used  to  train  "A"  school  level  students  to  maintain  those  electronic 
equipments  typically  maintained  by  electronic  technicians  (ETs)  and  electronic  warfare 
technicians  (EWs)  in  the  fleet.  These  equipments  include:  radar,  communication, 
electronic  countermeasure  (ECM),  and  electronic  support  measure  (ESM)  equipments. 

b.  Functional  (Training  Mission)  f  escription.  An  EEMT  system  is  intended  to  be  a 
generic  system  that  will  train  more  students  more  cost  effectively  and  to  greater 
proficiency  in  less  time  than  current  training  processes. 

The  EEMT  system  is  intended  to  provide  Class  "A"  school  electronics  trainees 
with  hands-on  practice  of  generic  preventive  and  corrective  maintenance  procedures.  The 
system  will  provide  self-paced  instruction  adaptable  to  the  skills  and  needs  of  individual 
trainees.  Immediate  feedback  will  be  provided  on  trainee  performance.  The  system  will 
also  provide  both  computer-assisted  and  computer-managed  instructions  such  that  trainee 
performance  skills  can  be  tested.  These  features  will  also  allow  instructor-level  personnel 
to  schedule,  monitor,  and  assess  the  training  process  and  intervene  when  necessary. 

Training  tasks  will  be  allocated  between  the  two-dimensional  (2D)  and  three- 
dimensional  (3D)  units  of  the  system  such  that  those  tasks  that  can  be  trained  sufficiently 
on  the  2D  unit  are  taught  there.  Only  those  tasks  that  require  3D  cues  and  manipulation 
of  switches  and  controls  will  be  trained  on  the  3D  unit.  Accordingly,  it  is  intended  that 
training  will  be  carried  out  predominantly  in  an  interactive  2D  format  with  the  3D  unit 
used  primarily  for  hands-on  training. 

c.  Hardware  Description.  Each  EEMT  system  trainer  will  consist  of  a  2D 
interactive  display  unit  and  a  3D  simulated  electronic  equipment  unit.  The  2D  and  3D 
units  will  be  capable  of  a  combined  or  attached  configuration.  The  2D  unit  will  consist  of 
a  processor  with  support  software,  instructor  terminal,  graphic  display  device,  adaptive 
display  device,  and  user  input  device.  The  3D  unit  will  consist  of  a  processor  with  support 
software,  a  support  frame  containing  simulated  generic  electronic  equipment  modules, 
and  simulated  test  equipment  test  modules,  instructor  terminal,  user  input  device,  and 
performance  feedback  devices.  When  in  a  combined  configuration,  a  data  transmission 
link  (cable)  will  be  used  to  attach  one  2D  unit  to  a  single  3D  unit. 

2.  Critical  T&E  Issues  (from  NDCP-Z0789-PN,  16  dune  1978). 

a.  Issues  to  be  tested  during  DT-I11: 

(1)  Are  various  subunits  ot  system  functionally  compatible  when  interfaced? 

(2)  What  is  the  programming  feasibility  of  effectively  driving  and  integrating 
hardware  and  software? 

(3)  What  is  the  adequacy  of  the  developed  data  base  and  its  application  to 
effective  teachware? 


A-3 


(4)  Can  the  EEMT  system  be  designed  to  safeguard  against  student  misuse 
leading  to  injury  of  personnel  and  equipment? 

b.  Training  Issues. 

(1)  Issue:  Will  EEMT  system  training  improve  the  transition  between  the 
electronic  fundamentals  and  equipment-specific  segments  of  ET  and  EW  electronics 
maintenance  training? 

Resolution:  T<5cE  will  be  used  to  assess  the  performance  of  "A"  school 
graduates  from  the  proposed  training  system  with  that  of  graduates  from  the  existing 
curriculum. 

(2)  Issue:  Can  EEMT  system  training  reduce  the  need  for  operational  prime 
equipment  in  electronic  equipment  maintenance  training? 

Resolution;  This  issue  will  be  addressed  indirectly  by  estimating  the  number 
of  prime  equipments  that  would  have  been  required  to  provide  an  equivalent  amount  of 
hands-on  practice  in  the  existing  curriculum. 

(3)  Issue:  Do  the  tasks,  skills,  and  knowledges  learned  on  the  EEMT  system 
generalize  to  the  larger  population  of  equipments  maintained  by  ETs  and  EWs? 

Resolution:  Transfer  of  training  can  be  assessed  by  observing  hosv  well 
students  can  perform  the  learned  maintenance  tasks  on  specific  operational  equipment. 

(4)  Issue:  Will  the  EEMT  system  ie3d  to  a  training  improvement  of,  at  least,  10 
percent  in  terms  of  reduced  training  time,  ar.c  increased  levels  of  student  proficiency 
with  no  increase  in  retraining  time  at  "O’  school? 

Resolution:  This  issue  can  be  verified  with  data  from  the  pilot  course,  the 
end-of-course  test,  and  the  "C"  school  questionnaires. 

(5)  Issue:  Will  the  EEMT  system  be  self-sufficient  so  that  it  can  be  maintained 
and  supported  by  school  personnel? 

Resolution:  The  ability  of  school  personnel  to  support  and  maintain  the 
EEMT  system  will  be  demonstrated  during  DT-IH. 

3.  Objectives  and  Thresholds 

a.  Program  Test  Objectives.  The  T<.VE  program  will  determine  if  the  device  can: 

(1)  Adapt  to  changing  job  demands. 

(2)  Accommodate  slow,  moderate,  and  fast  learners. 

(3)  Maximize  effectiveness  of  individual  hands-on  preventive  maintenance 

training. 

(4)  Increase  student/instructor  ratio. 

(5)  Train  to  appropriate  vacuum  tube,  solid  state,  and/or  large-scale  integration 
(LSI)  technology. 
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b.  Program  Thresholds 


(1)  Cost.  A  long-range  goal  is  to  provide  electronics  training  with  the  EEMT 
system  at  about  half  the  cost  needed  to  acquire,  operate,  and  maintain  conventional 
hardware  to  accomplish  the  equivalent  electronics  training.  Speciiic  recurring  design-to- 
cost  goal  is  $50,000  per  student  station  (2D-3D  unit  combination). 

(2)  Performance.  A  training  improvement  of  at  least  10  percent  shall  be 
required  to  continue  development.  Improvement  shall  be  expressed  in  reduced  training 
time,  reduced  reliance  on  operational  equipment,  increased  levels  of  student  proficiency 
on  related  training  system  measures,  and  no  increase  in  retraining  time. 

(3)  Schedule.  Delivery  dates: 


Item 

Date 

Serial  numbers  1  and  2  of  2D  EEMT  (NAVPERSRANDCEN) 

June  SI 

Serial  nurnDers  3  thru  12  of  2D  EEMT  (ET  "A"  School, 

Great  Lakes,  1L). 

August  81 

Serial  numbers  13  thru  20  of  2D  EEMT  (EW  "A"  School, 

Pen*  acola,  FL). 

December  81 

Training  Course  (Maintenance  of  2D,  M1L-STD  1379 

Training  Course). 

January  82 

Serial  numbers  1  and  2  of  3D  EEMT  (one  each  to  the 

ET  and  EW  schools  listed  above). 

April  82 

<>.  Required  Technical  Characteristics  (from  2D  EEMT  and  3D  EEMT  Specs.) 

Milestone  III  Post  111 

Parameter  Threshold  _ Goal 

2D  Unit 


Computer  system 


Language 

Pascal 

Ada 

Processing  capacity 

Worst-case  path 
<  50%  of  total 
time  for  iteration 
or  solution  cycle 

Same  as 
Milestone  III 

RAM  capacity 

<  50%  of  main 
memory  used  for 
program  and  data 
storage 

Same  as 
Milestone  III 

Disk  (mass  storage) 
capacity 

<  50%  of  total  used 
for  program  and 
data  storage 

Same  as 
Milestone  III 

Storage  capacity 

3.5  x  I08  bytes 

Same  as 
Milestone  III 
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Parameter 


Milestone  III 
Threshold 


Post  III 
Goal 


2D  Unit  (continued) 


Disk  (mass  storage) 
capacity  (continued) 

Access  speed  200 
ms/random  access 

Read  error  rate 

1  in  10s  bits 

Same  as 
Milestone  111 

Same  as 
Milestone  III 

Instructor  terminal 

Data  transmission 
speed 

300  baud 

Same  as 
Milestone  III 

Soft  vare 

Demonstration/ 

readiness 

Exercise  100%  of 
system's  functional 
components 

Same  as 
Milestone  III 

Graphic  (pictcral) 
display 

Resolution 

Capacity 

Display  image 
access  time 

TBD 

<  1 , >00  separate 
images  total,  or 

<  150  different 
background  images 

5  seconds  maximum 

<  50  images  acces¬ 
sible  in  <  2  seconds 

Same  as 
Milestone  III 

Same  as 
Milestone  III 

Same  as 
Milestone  111 

Same  as 
Milestone  111 

Image  distortion 

i%  of  maximum 
image  dimension 
minimum 

Same  as 
Milestone  III 

Adaptive  display 
device 

TBD 

User  input  device 

Precision 

Within  1/2  inch 
maximum 

Same  as 
Milestone  III 

Response  time 

2  seconds  maximum 

Same  as 
Milestone  III 

Mean  time  to  repair 
(MTTR) 

TBD 

Same  as 
Milestone  III 

Availability 

90% 

Same  as 
Milestone  III 
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Milestone  Ill  Post  III 

Parameter _  Threshold  _ Goal 

3D  Unit 

"Support"  mainframr 


Capacity 

Three  piug-in 
equipment  modules 

Six  plug-in 
equipment  modules 

Computer  system 

RAM  capacity 

<  50%  of  total 
used  for  program 
and  data  storage 

Same  as 

Milestone  III 

Disk  (mass  storage) 
capacity 

Transfer  rate 

Entire  operational 
program  of  data 
base  loaded  in  2 
minutes  or  less 

Same  as 

Milestone  ill 

Simulated  generic 
equipment  modules 

Expansion  capability 

100%  minimum 

None 

Software 

Demonstration/ 
readiness  exercise 
program 

All  functional 
components 

Same  as 

Milestone  III 

Mean  time  between 
failures  (MTBF) 

NA 

500  hours 

Service  life 

NA 

60,000  hours 

MTTR 

NA 

30  minutes 

Availability 

NA 

a  MTBF  it-Tn 

,9  x  TJTbP  +  MTTR 

3.  Required  Training  Characteristics  (from  NDCP-Z0789-PN,  16  June  1978,  2D  EEMT 
and  3D  EEMf  Specs.) 

a.  Overall  Training  Characteristics. 

(1)  Provide  hands-on  training  of  electronic  maintenance  tasks  and  principles, 

(2)  Provide  generic  systems  training;  that  is,  tasks  and  equipments  trained 
common  to  population  of  1  sks  and  equipments  used  by  F.Ts  and  EWs. 

(3)  Provide  individualized,  adaptive  training;  that  is,  provide  Immediate 
performance  feedback  and  accommodate  slow,  medium,  and  fast  learners. 

b.  Training  Effectiveness. 

(1)  A  50  percent  reduced  reliance  on  operational  prime  equipment  at  Milestone 
IU. 
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(2)  A  10  percent  performance  improvement  at  Milestone  Ill. 

Milestone  III  Threshold:  10  percent 
Post-Milestone  III  Goal:  TD3 

(3)  100  percent  of  training  objectives  for  2D  at  Milestone  111  met,  and  100 
percent  of  training  objectives  for  3D  met  at  post-Milestone  HI. 

c.  Hardware  Suitability. 

(1)  Reliability.  Design  reliabilities  for  the  2D  and  3D  units  are  given  in 
Required  Technical  Characteristics  (Part  II,  No.  4). 

(2)  Maintainability.  The  EEMT  system  will  be  maintained  by  Navy  personnel. 

(3)  Availability.  The  trainer  will  be  designed  to  be  available  for  scheduled 
operation  in  the  training  environment  a  minimum  of  nine -tenths  (0.9)  of  the  time. 

Availabilities  are  given  in  the  Required  Technical  Characteristics  (Part  II,  No.  4). 

(4)  Logistics  supportability.  The  EEMT  system  will  be  logistically  supportable 
at  the  schools  where  the  systems  are  deployed.  Spares  will  be  sufficient  to  repair  all 
critical/major  failures  during  a  (TBD)  month  period. 

(5)  Compatibility.  The  EEMT  system  will  be  physically,  functionally,  and 
electrically  compatible  with  the  school  laboratory  environments  in  which  they  are 
deployed. 

(6)  Interoperability.  Performance  data  from  the  EEMT  system  will  be 
interoperable  with  Navy  CMI  systems. 

(7)  Training.  Operation  of  the  EEMT  system  will  require  no  more  than  2  weeks 
of  formal  training  supplemented  by  on-the-job  training.  Maintenance  of  the  system  will 
require  no  more  than  4  weeks  of  formal  training  supplemented  by  on-the-job  training. 

(8)  Transportability.  Trainer  design  will  stress  ease  of  transportation  and 
simplicity  of  assembly  and  disassembly. 

(9)  Safety.  The  simulator  will  incorporate  basic  system  safety  principles. 
Safety  devices,  provisions,  or  procedures  will  be  provided  as  necessary  to  ensure  maximum 
safety  during  operation  and  maintenance  of  the  trainer.  The  provisions  of  Requirement  1 
of  MIL-STD  454  shall  be  used  as  a  design  code. 

(10)  Human  factors.  The  trainer  will  incorporate  man/equipment  interface 
features  that  maximize  efficient  and  safe  operation.  The  provisions  of  MIL-STD  1472  will 
be  used  as  a  design  guide. 

6.  Environmental  Impact  of  T&E 


None. 
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Electronic  Equipment  Maintenance  Training  System  (Device  11B106) 

Part  IV 

Device  Test  and  Evaluation  (DT&E)  Outline 

1.  DT&E  to  Date. 

The  range  of  training  objectives  required  for  the  device  has  been  identified.  The 
fidelity  requirements  have  been  resolved.  Alternative  conceptual  designs  have  been 
developed  for  the  EEMT  system.  One  alternative  has  been  selected  for  prototype 
development. 

2.  Future  DT&E. 

a.  DT-1H  for  the  prototype  development  will  be  carried  out  in  three  separate  phases 
during  FY81  and  FY82.  The  2D  and  3D  units  will  be  tested  separately  during  Phases  A  and 
B  respectively.  The  integrated  system  will  be  tested  during  Phase  C. 

b.  DT-I11  A  (FY82). 

(1)  Equipment  description.  Prototype  EEMT  system  trainer  2D  unit  hardware, 
software,  teachware,  and  documentation  will  be  required.  The  2D  unit  will  be  a 
"brassboard"  prototype. 

(2)  DT&E  objectives.  Testing  will  be  designed  to  exercise  the  features  listed  in 
Table  A-l  and  will  emphasize  teachware-trainee-display  interactive  features,  search 
time,  and  accuracy  of  displayed  graphics. 

(3)  DT&E  events/scope  of  testing/basic  scenarios.  Appropriate  testing  for  the 
2D  stand-alone  unit  using  programmed  teachware  moduies  will  be  conducted  during  DT-III 
A  at  the  contractor's  facility.  Testing  will  include  reliability,  maintainability,  and 
supportability  during  functional  use  over  a  3-month  period. 

The  teachware  features  will  be  tested  by  generating  and  verifying  a  brief 
teachware  module  from  a  prepared  script.  The  instructional  and  evaluation  feaures  will 
be  tested  by  running  through  teachware  modules  specifically  prepared  to  exercise  these 
features  and  by  comparing  the  system  responses  and  hardcopy  output  to  the  expected 
results.  The  remaining  features  will  be  tested  by  inspection  and  demonstration. 

(4)  Quantifiable  scope  of  effort.  Testing  will  require  90  days  of  effort  and  at 
least  two  teachware  modules  and  accompanying  scripts.  The  module  used  to  verify  the 
teachware  system  will  be  entered  into  the  computer  at  the  time  of  testing  from  a 
prepared  script.  The  remaining  teachware  module(s)  will  be  disk-resident  and  designed  to 
exercise  the  remaining  instructional  and  evaluation  features.  These  same  modules  will  be 
used  again  and  run  repeatedly  for  at  least  (TBD)  hours  a  day  for  (TBD)  consecutive  days  to 
evaluate  reliability,  maintainability,  and  supportability. 

c.  DT-I11  B  (FY82). 

(1)  Equipment  description.  Prototype  EEMT  system  trainer  3D  unit  hardware, 
software,  teachware,  and  documentation  will  be  required.  The  3D  unit  will  be  a 
"breadboard"  prototype. 
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Table  A-l 


2D  Unit  Features  to  be  Exercised  During  DT -III 


Unit 

Feature 

Computer  Hardware/Software 

Language- -Pascal  or  Ada 

CPU  processing  time 

Size  of  RAM  of  mass  storage 

Instructor  Terminal 

Interfaces  with  any  2D  unit 

Initiates  operating  modes 

Generates  and  modifies  teachware 

Runs  ail  programs 

Medium  to  low  speed 

Interface 

Passes  ASCII  characters 

Support  Software 

Computer  maintenance  and  utility  programs 
Demonstration/ readiness  program 

System  diagnostic  software 

Graphic  Display  Device 

Displays  computer-selected  static  images 

High  resolution 

5  seconds  maximum  access  time 

At  least  50  images  accessible  within  2  seconds 
Display  accuracy  at  least  1%  of  maximum 
image  dimension 

Interactive 
>  1500  images 

Adaptive  Display  Device 

Capable  of  computer  generated  guidance 
Interactive 

Provides  feedback  and  diagnostics 

User  Input  Devices 

Selectivity  precision-- 1/8  inch 

Audio  indication  of  entry 

Displays  selection  within  2  seconds 

System 

Portable 

Clearly  marked  test  points 

Interchangeable  and  standard  equipment  and 
parts 

Keeps  student  records 

Administers  performance  tests 

(2)  DT&E  objectives.  DT-iH  B  will  consist  of  analysis  and  functional 
demonstrations  to  verify  design  and  performance  characteristics  of  the  3D  unit.  The 
primary  objective  of  DT-III  B  is  to  test  the  feasibility  of  the  design  and  to  demonstrate 
the  implementation  concept  for  the  3D  unit. 

(3)  DT&E  events/scope  of  testing/basic  scenarios.  Only  testing  and  evaluation 
appropriate  for  the  3D  unit  operating  in  a  stand-alone  mot  connected  to  the  2D  unit) 
configuration  will  be  conducted  during  DT-III  B.  Testing  will  be  conducted  using 
teachware  and  documentation  prepared  specifically  for  that  purpose.  Because  of  the 
breadboard  nature  of  the  3D  unit,  it  will  not  be  necessary  to  conduct  extensive  tests  of 
reliability,  maintainability,  and  supportability,  etc.  Therefore,  it  will  be  possible  to 
conduct  all  testing  at  the  contractor's  facility  parallel  with  or  immediately  following 
acceptance  testing. 

The  basic  scenario  for  testing  will  be  the  same  as  in  DT-III  A  but  will 
exercise  the  features  listed  in  Table  A-2.  It  will  emphasize  trainee/simulated  hard¬ 
ware/simulated  test  equipment  timing  relationships,  and  visual/ functional  fidelity 
features. 

(4)  Quantifiable  scope  of  effort.  DT-III  B  will  require  (TBD)  days  of  testing  and 
evaluation.  At  least  two  teachware  modules  and  accompanying  scripts  will  be  needed. 
These  will  be  similar  to  and  serve  the  same  purpose  as  those  to  be  used  in  DT-III  A  but 
will  be  designed  to  exercise  the  features  given  in  Table  A-2. 

d.  DT-III  C  (FY82). 

(1)  Equipment  description.  Prototype  EEMT  system  trainer  2D  and  3D  unit 
hardware,  software,  teachware,  and  documentation  will  be  required.  This  is  the  same 
equipment  used  for  DT-III  A  and  B. 

(2)  DT&E  objectives.  DT-III  C  will  consist  of  analyses  and  functional 
demonstrations  to  verify  and  demonstrate  the  interactive  features  of  the  2D  and  3D  unit 
hardware,  software,  and  teachware. 

(3)  DT&E  events/scope  of  testing/basic  scenarios.  Only  testing  and  evaluation 
appropriate  for  the  EEMT  system  trainer  2D  and  3D  units  operating  in  a  connected 
configuration  will  be  conducted  during  DT-III  C  testing.  Testing  will  be  conducted  using 
teachware  developed  specifically  to  exercise  the  interactive  features  of  a  combined 
2D/3D  system.  Testing  will  focus  on  2D/3D  processor  communication,  2D/3D  data 
transmission,  and  integration  of  problem  exercises  requiring  alternate  use  of  the  2D  and 
3D  units. 

(4)  Quantifiable  scope  of  effort.  DT-III  C  testing  will  occur  (TBD)  following 
installation  of  the  2D  and  3D  units  at  the  ET  "A"  School,  Great  Lakes,  IL,  and 
immediately  preceding  commencement  of  DT-III  A.  It  will  require  (TBD)  days  of  effort 
and  at  least  two  teachware  modules  to  exercise  the  system's  interactive  features. 


Table  A-2 


3D  Unit  Features  to  be  Exercised  During  DT -III 


Unit 

Feature 

Teachware 

Interactive  authoring  language 

Conversational 

Menu  of  editing  functions 

Query  and  student  assistance  capability 

Can  generate  and  modify: 

•  Training  sequence 

•  Maintenance  procedures 

•  Simulated  readouts  and  waveforms 

•  Faults 

•  Performance  criteria 

•  Content  and  placement  of  cueing  and  feed¬ 
back  messages 

Accommodates  slow,  medium,  and  fast 
learners 

Performance  Evaluation 

Extrinsic  feedback 

Hardcopy  performance  summary 

Generic  Simulated  Equipment  Modules 

Specified  fidelity 

Functionally  realistic 

Can  be  configured  into  equipments 

Malfunction  insertion 

Instructor  Interface 

Connects  to  Support  Module 

Teachware  generation  and  modification 
Hardcopy  output 

Student  Input  Device 

Call  for  help 

Sign  on 

Select  and  load  exercises 

Identify  malfunctions 

Troubleshoot  and  isolate  faults 

Identify  faulted  unit(s) 

Indicate  repair  actions 

System 

System  readiness  checks 

Safeguards  against  trainee  error  and  misuse 
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Part  V 

Training  Capabilities  Test  (TCT)  Outline 


1.  TCT  to  Date 
None. 

2.  Future  TCT 


a.  TCT-I11  A  (FY81-82). 

(1)  Equipment  description.  Prototype  and  EEMT  system  trainer  2D  and  3D  unit 
hardware,  software,  teachware,  and  maintenance  and  operations  documentation  will  be 
required.  Other  supportive  media  and  materials  necessary  for  the  EEMT  system  trainer 
will  also  be  required.  This  is  the  identical  equipment  used  for  DT-III. 

(2)  TCT  objectives.  The  objective  of  TCT-II1  A  is  to  experimentally  evaluate 
both  EEMT  system  performance  and  trainee  achievement  when  the  entire  EEMT  system 
(trainer,  teachware,  and  supportive  materials}  is  used  in  the  ET  "A"  school  environment. 
Using  the  requirements  of  Part  II,  section  3A.(1)  (NDCP-Z07S9-PN,  16  June  1978)  as  the 
basis,  trainee  performance  will  be  assessed  to  determine  EEMT  effectiveness  as  outlined 
in  Table  A-3,  which  displays  the  major  EEMT  subsystem  components  to  be  integrated 
during  the  TCT  phase  of  the  program. 

(3)  TCT  events/scope  of  testing  basic  scenario.  For  TC7-I1I  A,  the  EEMT 
system  will  be  used  in  the  ET  "A"  school,  Great  Lakes  and  the  EW  "A"  school,  Pensacola. 
Following  installation  and  checkout  of  the  EEMT  system  trainer  at  the  two  designated 
schools  during  DT-III  C,  instructors  and  staff  who  will  use  the  EEMT  system  during  TCT- 
III  A  will  be  trained.  Additionally,  a  site  for  the  equipment  to  be  used  and  performance 
test  items  and  standards  for  the  end-of-course  performance  test  will  be  prepared  and 
finalized. 


A  split,  matched  group  design  will  be  used  for  TCT-III.  Two  groups  of 
subjects  (trainees)  of  at  least  50  subjects  per  group  will  be  selected  from  each  of  two  "A" 
school  pipelines  during  the  second  quarter  of  FY82  and  matched  on  existing  standard  test 
scores.  At  each  of  the  two  schools,  one  group  (control)  will  be  given  the  standard  "A" 
school  course  followed  by  the  end-of-course  performance  test.  The  two  experimental 
groups  will  take  the  same  curriculum  as  their  comparable  control  groups  but  with  the 
EEMT  system.  Identical  final  performance  tests  will  be  given  to  both  control  and 
experimental  groups.  An  analysis  of  these  data  will  provide  an  estimated  index  of  both 
training  and  cost  effectiveness  of  the  EEMT  system. 

(4)  Quantifiable  scope  of  effort.  TCT-III  A  will  require  50  ET  "A"  school 
students  in  each  of  two  groups  at  the  E'fcjass  "A"  schools  (toted  100).  No  less  than  30 
members  of  each  group  must  complete  the  planned  training  and  testing  sequence.  A  test¬ 
bed  for  the  performance  test  must  be  selected  and  secured.  The  end-of-course 
performance  test  and  an  alternative  form  of  this  test  will  be  needed  and  should  require  no 
more  than  60  minutes  per  trainee  to  complete.  EEMT  system  teachware  modules  and 
related  curriculum  materials  will  also  be  required.  The  EEMT  system  teachware  modules 
should  have  the  same  content  and  be  of  at  least  equal  quality  to  the  existing  curriculum. 


Table  A-3 


Major  EEMT  System  Test  and  Evaluation  Program  Phases  and  Integration  Components 


Phases 


1.  Equipment  Demonstration 


2.  Equipment  and 

Representative  Teach 
ware  Demonstration 


3.  Equipment,  Teachware 
Curriculum,  and  Training 
Utilization/Effective¬ 
ness  Demonstration 


2D  EEMT 
Subsystem 

3D  EEMT 
Subsystem 

EEMT 

System 

Hardware 

Hardware 

Hardware 

Software 

Software 

Software 

Hardware 

Hardware 

Hardware 

Software 

Software 

Software 

Representative 

Teachware 

Representative 

Teachware 

Representative 

Teachware 

Hardware 

Hardware 

Hardware 

Software 

Software 

Software 

Teachware 

Curriculum 

Teachware 

Curriculum 

/ 

Teachware 

Curriculum 

Representative 
trainee  popula¬ 
tion  (varying 
ability  levels) 

Representative 
trainee  popula¬ 
tion  (varying 
ability  levels) 

Representative 
trainee  popula¬ 
tion  (varying 
ability  (levels) 

Times  to  reach 

criterion 

performance 

Times  to  reach 

criterion 

performance 

Times  to  reach 

criterion 

performance 

Degree  of  trans¬ 
fer  of  training 
task  to  specific 
equipment 
training/job 
task 

Degree  of  trans¬ 
fer  of  training 
task  to  specific 
equipment 
training/ job 
task 

Degree  of  trans¬ 
fer  of  training 
task  to  specific 
equipment 
training/ job 
task 

b.  TCT-III  B  (FY82). 


(1)  Equipment  description.  Same  as  TCT-III  A  except  substitute  EW  "A"  school, 

Pensacola,  FL. 

(2)  TCT  objectives.  Same  as  TCT-III  A  except  substitute  EW  "A"  school, 
Pensacola,  FL. 

(3)  TCT  events/scope  of  testing/basic  scenarios.  Same  as  TCT-III  A  except 
substitute  EW  "A"  school,  Pensacola,  FL. 

(4)  Quantifiable  scope  of  effort.  Same  as  TCT-III  A  except  substitute  EW  "A" 
school,  Pensacola,  FL 

c .  Follow-up  Study  (FY82). 

(1)  Equipment  description.  No  equipment  is  required. 

(2)  TCT  objectives.  The  objectives  of  the  follow-up  study  are  to  assess  the 
performance  of  the  experimental  and  control  groups  (TCT-III  A  and  TCT-III  B)  during 
subsequent  "C"  school  and  fleet  activity. 

(3)  TCT  events/scope  of  testing/basic  scenarios.  The  follow-up  investigation 
will  involve  conducting  surveys  at  the  "C"  schools  and  fleet  assignments  to  determine  the 
comparative  performance  of  subjects  in  tne  control  and  experimental  groups.  Questions 
will  be  developed  during  the  third  quarter  of  FY82  and  sent  to  supervisory  personnel  at 
the  "C"  schools  and  fleet  billets  to  which  subjects  are  assigned  3fter  completing  "A" 
school. 

d.  TCT-1V  (FY82). 

Modification  and/or  corrections  will  be  made  as  required  to  meet  objectives  of 
TCT-III  A  and  B  (OPNAVINST  3S60.10,  page  6,  para  D  (d)). 

3.  Critical  T&E  Items 

a.  TCT-III  A  and  B. 

(1)  EEMT  system  teachware  modules  and  documentation  available  should  be  the 
same  content  and  of  at  least  equal  availability  to  existing  curricuiums. 

(2)  Instructors,  test  personnel,  and  analysis  and  simulation  support  people 
trained  at  ET  and  EW  "A"  schools. 

(3)  Standard  test  scores  obtained  and  control  and  experimental  groups  selected. 

(4)  Performance  tests  developed  and  documented. 

(3)  Operational  equipments  for  testing  selected  and  available. 


(1)  Survey  forms  developed. 

(2)  Analysis  support  people  trained. 

(3)  "C"  school  and  fleet  support  arranged. 
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